Tree-Structured Classifiers

A tree-structured classifier is a decision tree for predicting a class variable from one or more predictor variables. THAID [15, 7] was the first such algorithm. This article focuses on the CART R © [2], C4.5 [17], and GUIDE [12] methods. The algorithms are briefly reviewed and their similarities and differences compared on a real data set and by simulation. In a typical classification problem, we have a training sample L = {(X1, Y1), (X2, Y2), . . . , (XN , YN )} of N observations, where each X = (X1, . . . , XK) is a K-dimensional vector of predictor variables and Y is a class variable that takes one of J values. We want to construct a rule for predicting the Y value of a new observation given its value of X. If the predictor variables are all ordered, i.e., non-categorical, some popular classifiers are linear discriminant analysis (LDA), nearest neighbor, and support vector machines. (Categorical predictor variables can be accommodated by transformation to vectors of 0-1 dummy variables.) Although these classifiers often possess good prediction accuracy, they act like black boxes and do not provide much insight into the roles of the predictor variables. A tree-structured classifier (or classification tree) is an attractive alternative because it is easy to interpret. It is a decision tree obtained by recursive partitioning of the X-space. An observation in a partition is predicted to belong to the class with minimum estimated misclassification cost. Classification trees have been demonstrated to possess high prediction accuracy compared to many other methods; see, e.g., Lim et al. [11], Perlich et al. [16], and Loh [12]. They do not require categorical predictor variables to be transformed. THAID [15, 7] is the first published algorithm. We review here the CART R © [2], C4.5 [17], and GUIDE [12] algorithms and illustrate their similarities and differences on a real data set and by simulation.